Skip to content

feat(vortex-bench): infra of SpatialBench on DuckDB, plain binary#8598

Open
HarukiMoriarty wants to merge 4 commits into
developfrom
nemo/spatial-wkb
Open

feat(vortex-bench): infra of SpatialBench on DuckDB, plain binary#8598
HarukiMoriarty wants to merge 4 commits into
developfrom
nemo/spatial-wkb

Conversation

@HarukiMoriarty

@HarukiMoriarty HarukiMoriarty commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a SpatialBench benchmark that runs on DuckDB across two storage lanes, Vortex binary BLOB and Parquet binary BLOB.

Benchmark results

Local, warm, median of 5 (ms).

Q Query Parquet Vortex Winner
Q1 trips within 50km of Sedona 170.2 ms 132.7 ms Vortex 1.3×
Q2 trips in Coconino County 379.5 ms 249.2 ms Vortex 1.5×
Q3 monthly stats within 15km 235.5 ms 203.3 ms Vortex 1.2×
Q4 top-1000-tip zone distribution 404.8 ms 114.8 ms Vortex 3.5×
Q5 repeat-customer convex hulls 3174.8 ms 3187.3 ms tie
Q6 zone stats in bbox 718.8 ms 306.1 ms Vortex 2.3×
Q7 route-detour ratios 938.6 ms 1046.1 ms tie (~noise)
Q8 nearby pickups per building (500m) 1000.6 ms 1071.2 ms tie (~noise)
Q9 building conflation (IoU) 36.4 ms 40.6 ms tie
Q10 zone stats (LEFT JOIN) timeout timeout
Q11 cross-zone trips timeout timeout
Q12 5 nearest buildings (KNN) timeout timeout

Takeaways

Vortex wins the scan/filter-bound queries (Q1–Q4, Q6) by up to 3.5×. Q5/Q7/Q8/Q9 are compute-bound (DuckDB spatial join dominates), so the formats converge. Q10–Q12 time out on both, because the unindexed spatial join is the wall, independent of storage format.

@HarukiMoriarty HarukiMoriarty requested a review from a team June 25, 2026 20:14
@HarukiMoriarty HarukiMoriarty added the changelog/feature A new feature label Jun 25, 2026
@HarukiMoriarty HarukiMoriarty changed the title feat(vortex-bench): infra of SpatialBench on DuckDB, wkb only feat(vortex-bench): infra of SpatialBench on DuckDB, plain binary Jun 25, 2026

@myrrc myrrc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Can you please add support of this benchmark to vx-bench binary as well?

@AdamGS Can you have a look regarding your work of moving data generation to vx-bench?

threads: Option<usize>,
) -> Result<(Database, Connection)> {
let db = match path {
Some(path) => Database::open(path),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like these changes are already in develop, can you rebase?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I will rebase it.

@myrrc myrrc requested a review from AdamGS June 26, 2026 09:29
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@codspeed-hq

codspeed-hq Bot commented Jun 26, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.66%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 1588 untouched benchmarks
⏩ 4 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation slice_empty_vortex 339.4 ns 397.8 ns -14.66%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing nemo/spatial-wkb (c1635cb) with develop (3451cb0)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Comment thread vortex-bench/src/benchmark.rs Outdated
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@HarukiMoriarty

Copy link
Copy Markdown
Contributor Author

Fixed, wire to vx_bench in next PR.

@HarukiMoriarty HarukiMoriarty requested a review from a10y June 26, 2026 14:39
HarukiMoriarty and others added 2 commits June 26, 2026 13:38
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@HarukiMoriarty

Copy link
Copy Markdown
Contributor Author

@myrrc wire in vx_bench in #8607.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants